1. Field of the Invention
The present invention relates to a process of measuring the motion of a deformable object's surface and, more particularly, the present invention relates to a field of image recognition for measuring, in real time, facial shape from images obtained by a video camera, for example.
2. Description of the Background Art
Human facial expression is extremely important in the communicative interactions of humans. Similarly, human facial expression is important in the so called man-machine interface.
During face-to-face interaction, face motion conveys information at various levels. The transmitted information includes a person's emotional condition, position in a discourse, and while speaking, phonetic details about the speech sounds being produced.
Though various attempts have been made to analyze face motions from sequences of video images, most attempts either have not been based on quantitative evaluation or they have been restricted to limited quantitative evaluation of facial features, pre-determined by some combination of statistical and artistic techniques. Though such techniques can provide useful results, they often rely heavily on artistic rather than empirical techniques, they require tedious frame-by-frame examination, and they have not provided sufficiently accurate measures of face motion to be useful in coding dynamic attributes/properties of the face during, for example, spoken communication.
On the other hand, precise measures of three-dimensional locations of various portions of the face can be made using systems that track markers placed directly on the face surface. Such systems, however, have the following disadvantages.
(1) The equipment necessary for measurement is not general purpose but is highly specialized, thus limiting the scope and utility of applications. Further, the markers must be placed directly on the face, which is invasive for the subject. Therefore, use of such systems requires cooperation of the subject. A third limitation is that these systems cannot be used outside the laboratory environment.
(2) Since only a limited number of markers can be placed on the face, decisions about optimal placement of the markers to attain most effective and exact measurements must be made a priori. To date, however, it has not been possible to optimize the placement of the markers.
Another approach for measuring the motion of a person's face is to extract information quantitatively representing the face motion from sequences of video images. Video image recording requires only a video camera and recorder. Therefore, it is not physically invasive and can be used anywhere, thus overcoming the restrictions of marker-based measurement systems.
The approach using video images, however, involves difficulties such as the image distortion arising from conversion of the three-dimensional information to the two-dimensional information at the time of image pickup, the problem of poor spatial resolution and non-linear lighting effects. These problems are not limited to measurement of human facial expression; they apply to quantitative measurement of the surface motion of any three-dimensional object—animate or inanimate, rigid or non-rigid.